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HOW DOES A PRINCIPAL USE KENTUCKY’S HIGH STAKES ASSESSMENT 
TO MONITOR AND IMPROVE STUDENT LEARNING? 

By 

Ora Cobb, Jr., Superintendent 
Bellevue (KY) Ind. S.D. 

Jane Clark Lindle, Ph.D., Associate Professor 
& 

James S. Rinehart, Ph.D., Associate Professor 
University of Kentucky 

Objectives or Purposes 

This exploratory study encompassed two purposes. First, it used a forced entry regression 
model to detect which independent variables were predictors of success for “at-risk” students on 
the Kentucky Instructional Results Information System (KIRIS). Second, the study was intended 
to offer an analytic model for principals to use in examining the impact of specific school reform 
initiatives and selected intervention strategies. 

Three research questions shaped the data collection and analyses of this study. 

[1] Are the state mandated Family Resource and Youth Services Centers a 
significant predictor of student achievement for “at-risk” middle school students 
on the KIRIS assessments? 

[2] Are state/federally funded student remediation intervention strategies (i.e., 

Title 1 programs, Extended School Services programs, and summer remedial 
programs) significant predictors of student achievement for “at-risk” middle 
school students on KIRIS assessments? 

[3] Is student attendance a significant predictor of student achievement for 
“at-risk” middle school students on KIRIS assessment? 

Perspectives and Frameworks 

In general this study adopted the recommended practices of Effective Schools research 
that student test scores should be disaggregated and analyzed for data trends affecting at-risk 
students. In particular this study looked at aspects of the 1990 Kentucky Education Reform Act 
(KERA) and other known strategies for improving at-risk students’ scores to see if the strategies 
in one middle school had an effect on their standardized and state-performance-based assessment 
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results. This literature review discusses Effective Schools research and explains the background 
for KERA and the Kentucky state-performance-based assessment known as KIRIS, the Kentucky 
Instructional Results Information System. Finally it describes the basis for the specific strategies 
addressed in this school for improving at-risk students’ scores: Title 1, Extended School 
Services, Family Resources/Youth Services Centers, an advisory program, Saturday school, and a 
summer program. 



Effective Schools Research and Practices 

Effective Schools research of the late 1970s and early 1980s led to Effective Schools 
projects of the late 1980s (Brookover & Gigliotti, 1975; Cohen, 1982, 1981; Edmonds, 1979; 
Lezotte & Bancroft, 1985; Mace-Matluck, 1987; Pink, 1985; Purkey & Smith, 1983; Tomlinson, 
1981). While much of the literature focuses on the correlates of effective schools (Lezotte & 
Jacoby, 1991; Mace-Matluck, 1987; Purkey & Smith, 1982), in practice, schools are advised to 
disaggregate test scores to identify areas for improving instructional practices (Edmonds, 1982; 
Lezotte & Jacoby, 1991; Sudlow, 1984a, 1984b). This advice may limit practitioners’ adequate 
responses in at least two ways. 

One way this advice may be insufficient concerns the limitations of merely disaggregating 
scores. While attention to disaagregated test data is probably more analysis than many principals 
and teachers were prepared to do, disaggregation is only an initial step in addressing effects on 
test scores. Disaggregation may stimulate hypotheses on what actions should be taken to address 
sub-populations’ lower scores, but disaggregation does not necessarily establish relationships 
between instructional practices and test results. 

Another limitation to this advice concerns the nature of the scores in Effective Schools 
research. The test scores that Effective Schools reference are multiple-choice, standardized tests. 
Today many schools all over the U. S., but especially in Kentucky, are using performance-based, 
authentic assessments including portfolios and open-ended questions. Disaggregation of these 
scores may result in different hypotheses about addressing low performance among sub- 
populations. Still, disaggregation is only a preliminary tactic toward improving student and 
school performance. 

As part of the hypothesis testing process for any type of test, some instructional strategies 
must be applied to the sub-populations. In this study, we investigated some common strategies 
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for addressing the low performance of at-risk students. We identified a single middle school’s 
programs designed for its at-risk students. These programs included state and federally funded 
programs and, of course, local adaptations of such programs. Specifically the relevant federal 
program was Title 1. Then among several programs funded by the state due to the 1990 
Kentucky Education Reform Act (KERA), this study focused on Extended School Services 
(ESS) and Family Resource & Youth Services Centers (FRYSCs). Locally designed aspects of 
the state programs and a school level advising initiative were studied in relation to student 
performance. An understanding of the 1990 Kentucky Education Reform Act helps establish the 
context for this study as well as the dependent variable. 

Background on KERA and KIRIS 

The 1990 Kentucky Education Reform Act (KERA) was intended to address the severe 
deficits in Kentucky’s entire system of public education. These deficits were evident on every 
indicator of educational performance in the late 1980s, and the Kentucky supreme court used 
these deficits to declare the entire school system unconstitutional in a sweeping settlement of 
what had been a financial equity suit (Combs, 1991). 

The Kentucky General Assembly responded with a comprehensive education reform act , 
known as KERA, that mandated changes in governance, curriculum, and finance. The grounding 
for this investigation can be found in the juxtaposition of the mandates for changes in governance 
and curriculum. Although the state measures curricular results under KERA, schools serve as the 
authority for curriculum development. 

Part of the law required a new state-wide testing program that measures school 
performance in meeting Kentucky’s Learning Goals and Academic Expectations. Known as the 
Kentucky Instructional Results Information System (KIRIS), the assessment uses a combination 
of multiple choice, opened-ended responses and portfolios to determine school progress and 
allocates rewards to teachers (KRS 156.6453). Given the state of the art of measurement for 
authentic assessment, KIRIS has been investigated for its reliability (Catterall, Mehrens, Ryan, 
Flores & Rubin 1998; Petrosko, 1997). The investigations are due to criticism resulting from a 
highly charged political setting surrounding a high stakes assessment. Certified staff in schools 
deemed to have failed to improve at the rate established under KIRIS could potentially lose their 
jobs. Although to date, no one has lost his or her position, Kelley and Protsik (1996) found that 
the threat of sanctions was more potent for most staff than the actual receipt of rewards. 
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Into the mix of holding schools accountable was a presumption by the Kentucky 
legislature that certain autonomy should flow to schools (Task Force, 1989). Thus, under KRS 
160.345, schools operate School-Based Decision Making (SBDM) Councils to address at least 18 
broad policy issues (Russo, Donelan & Van Meter, 1993). Among those areas, curriculum is 
prominent. Schools not only determine curriculum policy and instructional practices, but they 
also select instructional materials and design professional development for teachers. Using 
KIRIS as the basis for making these curricular decisions represents a rational approach to 
improving student and school performance. Specifying particular programs for students with 
educational needs is one part of that approach. 

Title I 

Recognized as Chapter 1 or Title I, this federally funded student remediation program 
dates to the late 1960s. Originally designed as a program for disadvantaged children to increase 
their math and reading proficiency, Title I has undergone several incarnations (Reynolds, 1994). 
Generally, Title I results demonstrate increased achievement for participants (Eberhardt, 1992; 
Rotberg, 1993; Thistlethwaite & Mason, 1993). More recently, Title I and other school-based 
academic services have been combined with social services to students and their families (Adler 
& Gartner, 1994; Kirst, Koppich & Kelley, 1994; Macchiarola & Gartner, 1989; Wong & Wang, 
1994). Studies suggest that the combination of family services and remediation does benefit 
student achievement (Dill, 1993; Greene, 1993; Letgers & Slavin, 1992; Thistlethwaite & 

Mason, 1993). While Title I predates KERA, Kentucky added its own categorical programs for 
students at-risk in the form of Extended School Services (ESS) and Family Resource/Youth 
Services Centers (FRYSCs) 

Extended School Services, Saturday Tutoring & Summer School 

Extended School Services (ESS) is provided under KRS 158.070 to enable schools to 
“provide continuing education for those students who are determined to need additional time to 
achieve” (p.217). In general ESS programs emphasize intervention and remediation (Kentucky 
Teacher . 1995). 

School districts may design their ESS programs in a variety of ways. Schools have used 
before- and after-school sessions as well as Saturday and summer camps (Okorley & Drake, 

1997; Okorley, Lindle, Anderman & Jones, 1998). A few schools in Kentucky have adopted 
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Year Round Education in a 45-15 model of days of attendance and days off. During the 15 days 
off, some schools are running both enrichment and ESS programs. The dominant model for ESS, 
however, is an after-school program (Okorley, Lindle, Anderman & Jones, 1998). 

Although perceptions of ESS are positive and students exit with higher grades, other 
measures of student progress have been weak or nonexistent (Okorley & Drake, 1997; Okorley, 
Lindle, Anderman & Jones, 1998). To date, no comprehensive study of ESS effects on student 
performance have been undertaken. This study provides a glimpse at one school’s set of student- 
level effects. 

Family Resource/ Youth Services Centers 

The implementation of Kentucky’s model of interagency service delivery in schools rode 
the crest of political forces at local, state, and federal levels for collaborative attention to students 
and families at risk (First, Curcio & Young, 1994; Joe & Nelson, 1989; Russo & Lindle, 1994; 
Wong & Wang, 1994). The roots of such interagency collaboration were established in several 
federal programs such as Head Start and special education (Bronfenbrenner, 1979, 1986; Hobbes, 
1978; Kirst, Koppich & Kelley, 1994; Wang, Reynolds & Walberg, 1994). 

The implementation of Kentucky’s FRYSCs rolled out in stages. Beginning in 1992, 
schools with more than 20% of enrollees eligible for free or federally subsidized lunch applied 
for grants to establish centers. The phase-in began with 133 centers serving 232 schools in Fiscal 
Year 1992 and by 1996 supported 560 centers serving more than 900 schools (Wilson & Roeder, 
1997). One of the fundamental questions is whether the improved well-being of children and 
families will lead to improved student academic achievement (Firestone & Wilson, 1989; 
Fruchter & Price, 1993). At this writing, the connections between FRYSCs and student 
achievement have not been established, partially due to their relative recency (Prichard 
Committee, 1991). This study documents at least one school’s FRYSC program effects on 
student achievement. 



Summary of the Literature 

The purpose of this research was to explore the results of specific initiatives aimed at 
low-performing sub-populations on the state’s performance -based assessment in a Kentucky 
middle school. It used the suggested approaches for Effective Schools, but there are limitations 
to such advice. For example, Effective Schools research is founded on standardized test results; 
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this study utilized both standardized (CTBS) results and the KIRIS, a performance-based 
assessment designed for the state of Kentucky. This research had potential in supplying an 
analytic tool for principals engaged in high stakes assessment. Potential predictor variables were 
identified from school reform efforts directed at the target student population. Using a forced 
entry regression model, several of these variables were found significant in predicting student 
performance on KIRIS. 

Methods & Procedures 

After obtaining district, school, parents’ and students’ written consents, data were 
retrieved from individual students and specific, selected program records. New data were 
collected using student self-reports on the researcher-constructed School Reform Effectiveness 
Survey (SRES). 

Data Sources or Evidence 

Two types of data were used in this study: retrieved data and survey data. Retrieved data 
included Comprehensive Test of Basic Skills (CTBS) results, KIRIS scores, attendance, Title 1 
services, Extended School Services (ESS), summer school programs, and Family 
Resource/Youth Services Center (FRYSC) services. The survey data was collected through the 
researcher-made School Reform Effectiveness Survey (SRES). 

The SRES was developed to gather information about specific program initiatives as 
possible predictors of student success on the KIRIS assessment. Questions were developed in an 
open-ended format to allow students to indicate the number of times they participated in specific 
different FRYSC activities. The instrumentation was established through content validity. Two 
expert judges reviewed the questions for pertinence to the study and student understanding. 

The SRES asked students for information concerning the intensity of FRYSC services, 
frequency of phone and written communication between parents and teachers. The SRES also 
collected data on moderator variables (gender and age). For the purposes of this paper, the SRES 
data on number of FRYSC services, intensity of services, and student reports of the frequency of 
phone and written contacts were entered as a total score. These data were considered additive 
because each question asked for frequencies. Furthermore, the reliability of self-report data from 
at risk students on four separate measures could have confounded the results of the model. Thus, 
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self-report data were collapsed into one survey summary score. 

Two regression models were generated. One used the CTBS standardized test scores as 
the dependent measure. KIRIS total score was the dependent variable for the other model. 

Given the political environment for using KIRIS, the CTBS model was deemed necessary to 
confirm the KIRIS. 

The independent variables in each model were the same. Age was the chronological age 
reported by the student in years. Attendance was obtained from records and reported in percents. 
Coding of the Survey Summary for students’ self-reports of FRYSC activities was explained 
above. The reset of the variables were dummy coded (0,1). Generally, zeroes represented non- 
participation and ones represented participation, except for gender where 0=female and l=male. 
In alphabetical order, the independent variables were as follows: 

Age 

Attendance 

Extended School Services 
Gender 

Saturday Tutoring 

Summer Camp 

Survey Summary (FRYSC) 

Title 1 Programs. 

The eligible student group, defined as at-risk seventh graders, numbered 197. Twelve 
students were not present for the assessment. One student’s parents denied permission for study 
participation. Thus, the data set included 184 students. 



Results 

The results are reported by research questions 1 through 4. However, in the overall 
regressions for both the CTBS and the KIRIS, one predictor was found to contribute significantly 
to both equations: Extended School Services. Table 1 displays the results for the KIRIS Total 
Battery, and Table 2 shows the results for the CTBS. 

Insert Table 1 
About Here 



As depicted in Table 1, the model for the Total KIRIS was significant (R 2 =.46, F= 18.86, 




p< .01). Four predictor variables made significant (p < .05) contributions to the model: 
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attendance (/?= .19, p c.Ol); Extended School Services (/?= .45, p <.01); gender ((/?= -.15, p 
<.02); and Saturday tutoring (/? = -.13, p <.02). 

The contributions of attendance are probably not surprising. A logical assumption is that 
the more students are in school, then the more they learn or at least can score well on assessments 
they are taught. ESS is a welcome contributor to student results. This model suggests that ESS 
is meeting its designers expectations. The negative effects of gender and Saturday tutoring 
probably need some explanation and perhaps more investigation. In this case, gender was scored 
as a dummy variable with the lower code representing girls. These results confirm other studies 
which show that girls have scored better on KIRIS (Fayette County [KY] Assessment Office, 
personal communication, 1997). The negative weight on Saturday tutoring is a curious result. A 
cursory discussion with school officials in this single case revealed that the activities in the 
program may not have been strictly academic tutoring. Furthermore, the Saturday tutoring was 
not offered with enough lead time before the KIRIS testing window. Perhaps the negative effects 
are due to the weakest students’ assignment to Saturday tutoring compounded by a lack of 
academic focus. Surely, these results can be further investigated. 

The CTBS model yielded four predictors as well. The results can be viewed in Table 2. 



Insert Table 2 
About Here 



As seen in Table 2, the CTBS model using the same variables as the KIRIS model is also 
significant (R 2 =.38, F= 13.59, p< .01). This CTBS model also is partially confirmatory of the 
KIRIS model. Two of the CTBS model’s four predictors are also predictive of KIRIS and in the 
same direction. The four CTBS significant (p < .05) predictors are as follows: age (/?= -.15; p 
<.03); attendance (/?= .23, p c.Ol); ESS (/?= .28, p c.Ol); and Summer Camp (/?= .19, p c.Ol). 

The negative effects of age again relates to the at-risk nature of this population. Older 
students represent perhaps the neediest in terms of student achievement on standardized tests. As 
with the KIRIS model, attendance and ESS are significant predictors. The Summer camp result, 
while a welcome contributor to the model, is a bit of a surprise. The summer camp is run by the 
same people (FRYSC personnel) who run the Saturday tutoring program. Again, a cursory check 
with the school in this case revealed that summer camp was not particularly academically 
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oriented. The camp was focused on what might be considered enrichment activities in 
computers, library skills, art and music. To the extent that such general skills might be assessed 
in a standardized test the results is interesting and of course, warrants further research. 

Answers to the Research Questions 

Two models were generated using programs for at-risk to predict two kinds of assessment 
results: results on KIRIS, a performance-based assessment and CTBS, a standardized test. The 
models both yielded four overlapping predictor variables. These models were applied to answer 
the three research questions of this study. 

Research Question 1 : 

Are the state-mandated Family Resource and Youth Services 
Centers a significant predictor of student achievement for “at-risk” 
middle school students? 

In this study there were actually three variables entered into the regression models that 
applied to the FRYSCs: [1] Saturday tutoring, [2] summer camp and [3] Survey Summary — the 
aggregate of student reports of frequency of FRYSC services and parent-teacher contacts on the 
phone or in writing. The results were mixed in either model. For the KIRIS model, only 
Saturday tutoring was a significant predictor, but it was negative. For the CTBS model, only 
summer camp was a significant predictor. Both results suggest that either the variables were 
weakly measured and/or defined or that other means of investigating FRYSC effects on student 
achievement on any type of assessment deserves further investigation. Another explanation may 
be that students involved in FRYSC activities have more demanding and distracting issues to 
deal with at home or in the community than academic activities. 

Research Question 2: 

Are state/federallv funded remediation intervention strategies (i.e.. 

Title 1 programs. Extended School Services programs) significant 
predictors of student achievement for “at-risk” middle school 
students? 

The results were consistent for both the KIRIS and the CTBS models in this study. ESS 
was a strong predictor of student achievement, and Title 1 was not. ESS is operated as an after- 
school program which focuses on student performance and in many cases helps students prepare 
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for the state assessments. That ESS was a strong predictor for CTBS as well was a pleasant 
surprise. The results for Title 1 were not so pleasant, but also a surprise. In this school, Title 1 
has operated as a pull out program. Other research suggests that pull out assistance is not 
effective in Title 1 or special education (Wang, Reynolds & Walberg, 1994). In this school, and 
this is not an uncommon practice, weaker teachers were “retreaded” as Title 1 teachers. These 
results beg for further investigation of the differences between Title 1 and ESS in this case. 

Research Question 3: 

Is student attendance a significant predictor of student achievement 
for “at-risk” middle school students? 

Attendance was a strong and consistent predictor for both of the achievement models. 

The logic of high attendance producing higher achievement cannot be denied. In future research, 
perhaps the connect between programs for at-risk students and effects on attendance should be 
explored. 

Conclusions 

The investigation yielded two models which could be imitated for other schools and 
school councils in examining strategies for improving school results. The results from these 
models also emphasized unique characteristics of the variables embedded in the context of this 
school. Some of the variables yielded interesting findings that might not be replicated at other 
schools. 

The results showed a surprising contrast between two remedial programs, Title I and ESS. 
In this school, Title I was a rest stop for poorly motivated or marginal teachers. On the other 
hand, ESS has been tightly aligned with curriculum and assessment. 

The negative relationship with Saturday tutoring is also an unexpected result. Yet, the 
school revealed that Saturday tutoring was a “last chance” effort to prevent retention in grade. 

The program did not start until after the KIRIS assessment window had closed. It was not a 
preventive program and was not tied to student achievement on either the CTBS or the KIRIS. 

Because each of these programs were implemented in a fashion unique to this school, 
odds are high that these results will not be duplicated in another school. However, the model is 
useful as an example of what other schools might do in addressing school improvement and 
student achievement. 
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Significance of the Study 

This study was a first step in applying recommended practices, derived from Effective 
Schools research (Achilles, 1983; Bedford, 1988; Edmonds, 1982; Lezotte & Jacoby, 1991; 
Shoemaker & Fraser, 1981; Wynne, 1981), to a Kentucky middle school engaged in systemic 
reform in a high stakes, performance assessment environment. As a single case study, it 
represents an exploratory result that may be replicated in other similar settings and contexts. 
This study adds to the emerging literature on Kentucky’s comprehensive reform policies (e.g., 
Lindle, Petrosko, and Pankratz, 1997). This study makes an analytic contribution to the 
discussions of linking theory and data to practice. 
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Table 1 



Regression of KIRIS Index over Title I. age, attendance, gender, extended school services, 
summer camp. Saturday tutoring, and perceived services 
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F 
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Age 


-3.79 


2.23 


-.11 


-1.70 


.09 


.46 


18.86 


<.01 


Attendance 


.81 


.27 


.19 


3.03 


<.01 








Extended School 
Services 


26.35 


3.42 


.45 


7.71 


<.01 








Gender 


-8.27 


3.42 


-.15 


-2.42 


.02 








Saturday Tutoring 


-11.16 


4.90 


-.13 


-2.28 


.02 








Summer Camp 


6 .22 


4.37 


.09 


1.43 


.16 








Survey Summary 


- .06 


.15 


-.03 


-.44 


.66 








Title I Programs 


-4.54 


4.12 


-.06 


- 1.10 


.27 
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Table 2 



Regression of CTBS Scores over Title I. age, attendance, gender, extended school services, 
summer camp. Saturday tutoring, and perceived services 
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-5.33 


2.41 


-.15 


-2.21 


.03 


.38 


13.59 


<.01 


Attendance 


.97 


.29 


.23 
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-1.42 
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-1.37 
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Summer Camp 
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4.71 
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2.90 


<.01 








Survey Summary 


.01 


.16 


.004 
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-5.43 


4.44 
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-1.22 
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